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SECTION 1 - REQUIREMENTS FOR SOFTWARE RELIABILITY 

Digital flight controls are assuming an Increasingly Important role In the 
operation of passenger aircraft. At present they are essential for only some 
flight phases, such as automatic landing. But greater dependence on automatic 
stabilization can provide Increased fuel economy and speed, and thus there Is a 
need to provide freedom from failures (Including software failures) throughout 
the flight regime. These developments are already quite visible In military 
aircraft where automatic stabilization Is essential for safety of flight In 
major portions of the flight regime. 

Aircraft components essential to safety of flight are subject to FAA 
regulations, and compliance with these must be established In order to obtain a 
Certificate of FI Ightworthlness. As automatic flight controls become essential, 
they must comply with these requirements, the most Important one of which Is 
that any failure condition that would be catastrophic (to the aircraft) Is 
extremely Improbable CFAA703. An advisory circular equates 'extremely 
Improbable' with an expected failure frequency of IE-9 per flight or flight hour 
(according to some Interpretations this may require a failure frequency of less 
than IE-10 per hour). 

Since compliance must be demonstrated during a period when at most several 
thousand hours of flight time can be accumulated, demonstration of adequacy on 
the basis of observed failure frequency Is completely Impossible. For the 
hardware components, established methods of quantitative reliability analysis 
and fault tolerance are used to satisfy the requirements. The basis for such 
procedures In the software area Is much more problematic, and a purpose of this 
report Is to direct attention to Improvements In analysis and software systems 
synthesis that may overcome the existing deficiencies. Present software for the 
limited extent of essential applications Is usually certified by demonstrating 
that only a very restricted set of potentially catastrophic malfunctions may 
occur, and that there are checks or circumvention programs provided to guard 
against these. This practice Is not believed to be applicable to future broader 
app 1 1 cat Ions of digital f I Ight control s. 
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SECTION 2 - SCOPE AND DATA SOURCES 


It Is Intended to describe the state of software reliability for digital flight 
controls In the recent past and at present In quantitative and quall+atlve 
terms, to Identify trends, and to point to approaches that promise further 
Improvements and that may lead to a comprehensive certification methodology. 
Because very few reports on the reliability of flight control software have been 
published, data from a broader field of software applications Is examined, and 
the few available datapolnts from the flight controls field are discussed In the 
context of the general findings and trends. The present study Is divided Into 
two parts: a numerical evaluation of reliability and reliability trends, and an 

Investigation of causes of software failures that has both quan+ltatlve and 
qua I Itatlve aspects. 

The emphasis Is on presenting findings that may lead to Improvements In: 

Software development methodology 
Test methodology 

Application of software fault tolerance 
Quantification of software reliability 
Data collection and analysis 

The data sources utilized and some of Ihelr characteristics are summarized In 
Table 1. The first three entries In the table provide data for the numerical 
reliability assessment, and the last two entries together with the first one 
provide data for the evaluation of causes of software failures. The total 
database represented by these comprised 47 programs. Data for 28 programs In 
that set were sufficiently complete to permit the detailed analysis presented In 
this report. Significant characterl sties of these programs are summarized In 
the Appendix (Table A). 

TABLE 1 - DATA SOURCES 


Source 

No. of. 
Programs 

Program 

Size 

Fault 

Density# 

Fault 

Types 

Ames ( Set 1 ) 

15 

391k 

0.64$ 

Yes 

Ames/ (Set 2) 

2 

89k 

0.52$ 

No 

Goddard-SEL 

11 

812k 

0.10$ 

No 

Langley/MIPS 

1 

90k 

N/A 

Yes 

RADC/TRW 

3 

> 1,000k 

N/A 

Yes 


# Faults per equivalent executable assembly statement 

The first entry comprises data from software development for the B-1 bomber and 
the air launched cruise missile (ALCM). The data were collected for NASA Ames 
and Include some flight control and closely related programs (air data computer 
and navigation) [PRES81U. The Set 2 data were also collected for NASA Ames In a 
special effort to assist In the Identification of failure modes In digital 
flight controls CR0CK81]. The t&o programs relate to automatic flight controls 
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and aircraft navigation, respectively. The NASA Goddard Software Engineering 
Laboratory (SEL) data were acquired partly from the agency Itself and partly 
from the Data and Analysis Center for Software <DACS) at the Rome Air 
Development Center (RADC). These programs deal primarily with satellite 
telemetry and ground control of satellites [BASI77, CARDB2, TURNS! 3. The fourth 
entry comprises data collected under a NASA Langley contract during the 
development of the software for the Metric Integrated Processing System (MIPS) 
and Vandenburg Air Force Base. The specific program generates sensor 
calibration and other Initialization parameters for a missile launch monitoring 
application [HECH77H. The last dataset comes from a report prepared by TRW for 
RADC. The programs deal with satellite and missile applications [THAY76]. 

Throughout this report program size Is expressed In equivalent executable 
assembly statements. For programs written In most high order languages (HOLs) 
the number of executable statements Is multiplied by five to arrive at the 
assembly equivalent. This represents a conservative assumption for the 
hypothesis that the fault density In HOL programs Is lower than In assembly 
programs that will be presented below. Analyses of the expansion on a CDC 7600 
FORTRAN compiler Indicate an expansion factor of eight [LAWS76]. For programs 
written In AED a conversion factor of three Is used which Is derived from a 
specific analysis of the programs Involved here. Practically all data come from 
formal test programs, I. e., from the time between unit or module test and 
operational use of a program. 

By combining data from several sources some biases are probably Introduced Into 
this study due to differences In failure reporting, thoroughness of testing, and 
maturity of programs when data collection terminated. Some auxiliary analyses 
are therefore carried out within a given sample to determine whether the overall 
findings hold. However, the ensemble of the programs provides a good basis for 
studying broad trends which are our primary objective. 

In order to address the FAA requirements It Is desirable to express software 
reliability In terms of failure rate (failures per unit execution time) which 
then can be compared with the numerical criterion In the advisory circular. 
This measure was not available for the programs of primary concern In this 
study. The readily obtainable numerical Index for reliability was the fault 
density (faults per equivalent executable assembly statement). This Is the 
quantity used as the reliability measure In this report, usually expressed as a 
percentage. By means ot an *uncovery factor* (the number of executions required 
to detect a fault) the fault density can In theory be converted to a failure 
rate but there are at present no generally accepted values for the uncovery 
factor [MUSA79]. 

Standard definitions of reliability terms are used In this report [JEEE82]. 
Specif leal ly: 

e!5 

Fal I ure - The inability of a system or component to perform a required function 
within specified limits. A failure may be produced when a fault Is encountered. 

Ea,U 1 1 “ An accidental condition that causes a functional unit to fall to perform 
Its required function. 
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Error - A discrepancy between a computed, observed, or measured value or 
condition and the true, specified, or theoretically correct value or condition. 

Thus, a fault causes a failure which Is usually observed because of the error 
that results. In the Investigation of causes of failures we will discuss fault 
£fl±figOElfis although the looser term 'error category' Is also recognized In 
C IEEE82U, The relationship between these terms is Indicated In Figure 1. An 
example of the application of these concepts to digital flight control software 
may be represented by the following: an attitude calculation routine lacks 
protection against division by zero (this Is the fault); a zero divisor Is 
encountered (this Is the trigger); and this results In an Incorrect attitude 
output (the error). The coincidence of the fault and the trigger for that fault 
causes the failure. Reliability studies are concerned with the recording and 
avoidance of failures, and, as mentioned above, the preferred Index of 
reliability Is the failure rate. Since It was not possible to compute this for 
most of the programs In the data base, the analysis presented In the following 
section makes use of the fault density which represents a rough relative Index 
of reliability. 
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The trend of fault density with year of program start for all software from the 
first three sources In Table 1, Is shown In Figure 2. The fault density for 
each start year represents the total number of faults divided by the total 
number of statements In the corresponding programs. It Is thus a weighted 
average rather than the mean of the fault densities of Individual programs. The 
weighted average Is used In all comparisons that Involve groups of programs 
throughout this report. The trend curve Is also fitted by weighting each data 
point with the associated number of statements The faults that are considered 
here are those that resulted In errors or Improper program execution. In some 
studies the number of software problem reports (SPRs) Is used as an Index of 
software quality. Since a problem report may be generated to require a 
documentation change, or to add a feature now desired by a user but not part of 
the original requirements, the total number of SPRs Is not a good metric for 
reliability studies. 

Because of the wide scatter of the points, no statistical measures of 
significance are appropriate. However, for programs started In 1977 the fault 
density Is seen to be considerably reduced. The period between 1975 and 1977 
saw much Increased emphasis on software engineering (a disciplined approach to 
software development) and on the use of software tools (software programs that 
assist In the development or test of other programs). Also, HOLs came Into much 
wider use aurlng that period, and the effect of this Is discussed separately 
below. 

The contrast between pre-1977 software and more recent programs Is further 
examined In Table 2. Although there Is a considerable overlap In the range of 
fault densities observed for the to starting periods, there Is an obvious trend 
*‘0 reduced fault density for the recent starts. The fault density of the flight 
control programs Is below that of the whole group for pre-1972 starts but 
appreciably above the group average for the recent starts.’ Reasons for this 
latter deviation are discussed In connection with language use below. In either 
period the flight control programs are well within the range of the other 
programs. 


TABLE 2 - EFFECT OF STARTING PERIOD ON FAULT DENSITY 


Program 

Attribute 

Pre-1977 

Recent 

No. of programs 

7 

21 

HOL usage 

14$ 

71$ 

Program size* 

67k 

1 ,224k 

Fault density 

1.59$ 

0.22$ 

FI Ight controls 

1.19$ 

0.72$ 

Range of f. d. 0,63$ - 11.66$ 

0.01$ « 5.21$ 


* ^ u I valent executable assembly statements 


FAULT DENSITY, PERCENT 
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The great Increase In the use of HOLs during the recent period could be the 
significant cause for the Improvement In reliability. Therefore a separate 
analysis of the effect of starting period on assembly programs Is presented In 
Table 3. Although the reduction In fault density for recent starts Is not as 
pronounced as for the total sample In Table 2, It Is still present and can be 
seen In both the average and the range. Thus, the effects of disciplined 
software practices seem to have carried over even Into assembly language 
programming. A similar comparison for HOI programs could not be constructed 
because only a single HOL program had been started prior to 1977. 

TABLE 3 - EFFECT OF STARTING PERIOD ON FAULT DENSITY OF ASSEMBLY PROGRAMS 


Program 

Attribute 

No. of programs 

Program size* 

Fault density 

Range of f* d. 


Pre-1977 


6 

34k 

2 . 20 * 

0.63* - 11.66* 


Recent 


6 

100k 

1.03* 

0,15* - 5.21* 


*Executable assembly Instructions 

It Is also of Interest to examine the effect of language within a given time 
period. For this purpose the 21 programs that were started In 1977 and later 
are analyzed In Table 4. It Is seen that HOL usage does Indeed account for a 
very major reduction In fault density. The standard deviation of fault density 
among the HOL programs Is 0.21, The difference between assembly and HOL 
programs Is therefore In excess of four standard deviations (p < 0.005), The 
Inequality presented In parentheses Is an abbreviated notation to denote that 
the probability that such a large difference Is due to chance events Is less 
than 0.005. This notation Is used repeatedly In this report to Indicate the 
statistical confidence In selected findings. 


The flight control programs have a higher than average fault density In each 
language classification, and particularly for the HOL group where their average 
Is almost 2 standard deviations above that for the group as a whole. The two 
HOL flight control programs Involved here are written In AED and were started In 
1977 and 1978. Both the early starting date and the use of a language for which 
few program and test support facilities exist may account for the deviation. 
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TABLE 4 - EFFECT OF LANGUAGE ON RECENT PROGRAMS 


Program 

Attribute 

Assemb I y 

HOL 

No. of programs 

6 

15 

Program size* 

100k 

1,124k 

Fault density 

1.03$ 

0.15$ 

FI Ight controls 1.58$ 

0.52$ 

Range of f. d. 

0.15$ - 5.21$ 

0.01$ - 0.66$ 


* Equivalent executable assembly statements 

As mentioned earlier, there Is concern that the conclusions drawn from these 
comparisons might be biased due to the combination of data from several sources. 
The Goddard/SEL programs are all written In HOL and are In the recent group. 
These programs ere also at the low end of the fault density range. Within the 
programs from this source there Is no significant trend with respect to starting 
date or HOL use (some programs Include assembly sections but these are typically 
less than 10)? of the total code). To remove possible bias due to the 
combination of sources, the Ames (Set 1) data are analyzed separately In Table 
5. Although this Is a smaller sample, the same effects are apparent, and thus 
these do not appear to be artifacts Introduced by combining data from several 
sources. (The figures In parentheses under each entry In the table represent the 
number of programs and the total size). 

TABLE 5 - FAULT DENSITY FOR AMES (SET 1) PROGRAMS 


Starting 

FAULT DENSITY IN PERCENT 

Per I od 

Language: Assembly 

HOL 

Both 

Pre-1977 

2.20 

0.96 

1.59 


(6, 34k) 

(1, 33k) 

(7, 67k) 

Recent 

1.04 

0.175 

0.44 


(6, 100k) 

(2, 224k) 

(8, 324k) 

Both 

1.33 

0.276 

0.638 


<12, 134k) 

(3, 257k) 

(15, 391k) 


It Is seen that the reduction In fault density for recent starting dates and use 
of HOL carries over not only for this sample as a whole but also for each 
subgroup within It (HOL effects for each starting category, starting date 
effects for each HOL category), thus lending further credence to the conclusions 
stated earlier. From the analysis of the fault density data It appears that use 
of high order languages has a very pronounced beneficial effect on software 
reliability, and this finding can be translated directly Into the selection of 
the programming environment for flight control software and for similar critical 
applications. The effect of starting dates can be Interpreted as reflecting 



disciplined software development (structured programming, hierarchical 
structure, small module sizes, etc.) and better test technology, but It may also 
be due to some other factors. Further studies of this effect are warranted. In 
the meantime, we can be satisfied that whatever we may have been doing during 
the past five years In our approach to software development hcs proved to be 
beneficial. 

For a number of programs the data sources also provided Information on the 
professional labor hours expended during development. The following analysis Is 
restricted to 13 programs written predominantly In a H0L y and most of these come 
from the Goddard/SEL source. Four of these programs present pre-1977 starts, 
and the remaining nine were started more recently. A plot of fault density vs. 
programming hours per statement Is shown In Figure 3. The heavy line represents 
the weighted (by program size) linear regression for the entire population. The 
lighter line Is the linear regression for the recent programs only. Although 
the scatter Is wide and other factors obviously have a strong effect on fault 
density, there Is a pronounced negative trend (I. e., high programmer effort 
reduces the fault density). For both populations there Is extremely high 
confidence that the slope Is negative (p < 0.001). It would be Interesting to 
test whether this relationship still holds for effort above some threshold (e. 
g., above 2 hours/statement) but the amount of data available was not sufficient 
for a meaningful analysis. 


4. FAULT CLA&i NATION 

In spite of the encouraging results reported In the previous section. It must 
also be admitted that the overall results of the fault density analysis leave us 
far short of the reliability goals for essential flight controls. In order to 
apply remedial efforts effectively, the causes of faults In some of the software 
Identified In Table 1 will now be examined. 

A classification of causes of software faults was generated In a 1976 RADC 
sponsored study that examined several very large software programs [THAY76]. 
Because of the considerable effort that went Into that study, and because fault 
distributions that were published In It presented a unique basis for comparison, 
this classification (or adaptations of It) have been carried over Into most of 
the succeeding literature that examined software faults. Unfortunately, these 
classifications are not very satisfactory for describing software malfunctions 
In the context of FAA certification requirements where a classification by 
effect (error as defined above) would be more significant. Also, to judge the 
potential for catastrophic outcomes. It Is desirable to J»now the purpose of the 
routine that the fault Is located In (e.g., a fault In a Scheduler has greater 
likelihood of serious consequences than a fault In ani Internal recordkeeping 
module). None of this Information Is available In either the data specifically 
examined here nor In any other data source that has come to the attention of the 
authors. It Is also noted that the Identification and separate analysis of 
flight control software data represents a unique aspect of the present report. 

In spite of these shortcomings, the existing classification scheme provides 
Insights Into the deficiencies In the programming environment that lead to 
faults, and by Inference therefore points to avenues for eliminating causes of 
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CIRCLED POINT: PRE-1977 STARTS ALL OTHERS: RECENT (1977 OR LATER) STARTS 
LINEAR REGRESSION, RECENT STARTS ONLY 

•— linear regression, total population 

FIGURE 3 - EFFECT OF PROGRAMMING EFFORT ON FAULT DENSITY FOR HOL PROGRAMS 
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faults. The data collected specifically for this study provided ten major 
categories and additional subcategories which were utilized only to a limited 
extent. The original categories are shown In the left side of Table 6. To 
permit comparison with other data collections, some categories were combined as 
shown In the right column of the table. All of the following discussion will be 
In terms of the revised classification. 


TABLE 6 - FAULT CLASSIFICATION 


Original Classification 

A. Computational errors 

B. Logic errors 

C. Data Input errors 

D. Data handling errors 

E. Data output errors 

F. Interface errors 

G. Data definition errors 

H. Data base errors 

J. Other errors 

K. Unknown 

Over 3600 faults from 22 programs were classified In this manner. Some, but not 
all, of these programs were those for which fault density distributions were 
discussed above. Three of the programs were related to digital flight controls. 
The results of this classification are shown In the first three numerical 
columns of Table 7. The remaining columns of this table contain equivalent data 
from other projects. The MIPS program comprises about 25k FORTRAN statements, 
dealing with launch data preparation and calibration for a missile tracking 
system. The program was developed between 1975 and 1977 but Includes some 
portions from earlier programs. The TRW data come from the previously 
referenced RADC report. TRW 2 comprises 97k of JOVIAL J4 code dveloped In the 
early 1970s and Is Intended for batch processing. TRW 3 consists of over 115k 
JOVIAL J4 statements, likewise Intended for batch processing, and developed 
slightly later than TRW 2. A significant fact about TRW 3 Is that about 
one-quarter of the code was supplied by other contractors. TRW 4 Is a general 
Information processing system that can operate either on-line or In batch mode. 
It Is written In a macro-languge, PWS, and the exact size Is not stated. It Is 
a large program, consisting of 190 different routines. 

The flight control software data are presented In three columns In Table 7. The 
first of these Is labeled ’raw' and contains data as reported. The second 
column (AdJ. 1) contains an adjustment which transfers data scaling faults from 
the Data I/O category to Computational. In the third column a further 
adjustment Is made to move faults related to flags from the Data I/O to the 
Logic classification. The reason for those adjustments Is explained In the 
discussion of the table. 


Revised Classification 

a. Computational errors (A) 

b. Logic errors (B) 

c. Data I/O errors (CtDtE) 

d. Interface & execution errors (FtJ) 

e. Data base/global variables (GtH) 
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TABLE 7 - RESULTS OF FAULT CLASSIFICATION 


PERCENT OF ALL CLASSIFIED FAULTS 


Class 

Ames 

Total 

; (Set 1) 

FI Ight Controls 
Raw AdJ. 1 AdJ. 2 

Langley/ 

MIPS 

TRW 2 

TRW3 

TRW4 

a. Computation 

7.4 

10.0 

17.3 

17.3 

4.2 

17.2 

11.5 

2.4 

b. Logic 

47.1 

26.7 

26.7 

35.6 

66.3 

27.2 

30.5 

47.5 

c. Data I/O 

15.8 

32.3 

25.0 

16.0 

10.7 

16.6 

23.4 

12.2 

d. Interf. etc. 

16.2 

14.5 

14.5 

14.5 

15.6 

20.1 

20.2 

25.8 

e. Data base 

13,5 

16.5 

16.5 

16.5 

3.3 

18.8 

14.4 

12.2 


Comparisons among the elements of a row show sizeable differences (e. g., the 
percentage of computational errors ranging from 2.4? to 17.3$) but the projects 
as a whole show surprisingly good agreement In the distribution of faults among 
the classifications. With the exception of the ’Raw* flight controls programs 
from the present sample, logic faults are the largest class. DaTa I/O errors 
are mostly In the second or third largest category. 

In the ’raw* classification of faults for the Ames Flight Control programs, 
scaling errors were assigned to the Data I/O category. In most of the other 
data sources such errors are considered under computation (’Units Conversion 
Error* In CTHAY763K In the column headed ’Flight Controls, AdJ. 1* these 
faults (Incorrect scaling) were transferred from the Data I/O to the Computation 
classification. Even after this adjustment. Data I/O faults constitute an 
unusually high percentage, and logic faults an unusually low percentage, of all 
faults. A major contributor to this Is a large fraction of flag setting faults 
(flags not reset, not properly set, etc.). There Is precedent In CTHAY76] for 
classifying these as Data I/O faults but It must be recognized that In terms of 
tools and methodology that can be utilized to uncover these they are closely 
related to logic faults. Usually Incorrect handling of flags does not 
constitute a very large fraction of the faults, and this possibly Inconsistent 
classification does not present a problem. When flag related faults are 
classified as Logic (In the AdJ. 2 column), the pattern becomes much more 
consistent with that observed In the other data collections. Further 
Investigation of the high percentage of flag related faults In this data set 
appears warranted. Interface and execution errors show considerable uniformity 
among all the data sources. 

The total sample compiled as part of this study (first numerical column) shows a 
distribution ?jmong the fault classifications that Is well within the range of 
those of the previously reported efforts (all of which had pre-1977 start dates, 
with most representing 1970 - 75 starts). There appears to be no basis for 
attributing changes In the distribution of faults among the categories to 
changes In the programming environment. 

The total sample contained five programs that were written In a HOL, four that 
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were mixed (between 25$ and 75$ assembly), and 12 that were primarily written In 
assembly language. The fault classification for these three language groups Is 
shown In Table 8. The percentage of computational faults Is greater In the HOL 
programs which might be expected because these frequently deal with more complex 
algorithms. The percentage of logic faults Is greater In mixed and assembly 
programs but the Increase Is not of a magnitude that would Indicate a definite 
trend. The selection of language does not seem to have a major effect on the 
distribution of faults among the major classifications. 


TABLE 8 - FAULT CLASSIFICATION BY LANGUAGE 

PERCENT OF ALL CLASSIFIED FAULTS 


Class 

HOL 

Mixed 

Assembly 

a. Computation 

11.6 

6.9 

5.9 

b. Logic 

43.7 

47.9 

47.4 

c. Data I/O 

17.3 

17.0 

15.0 

d. Interf. etc. 

15.9 

14.4 

17.4 

e. Data base 

11.4 

13.9 

14.2 


The limitations of this classification must be acknowledged. It Is widely 
recognized that mistakes made In the very early phases of the development 
process are the most costly ones to remedy and frequently have the most severe 
consequences [B0EH763* Hence, Identification of the development phase during 
which the fault originated should be an Integral part of a fault classification 
scheme and use of such Information In subsequent analysis should be extremely 
useful. Because of the Judgement that Is Involved In this assessment most 
Investigators have not pursued this subject. It was attempted to supply this 
Information during the collection of some of the data analyzed here but this 
took place four to five years after the fault had been detected and did not 
produce consistent results. Data on the origin of faults (by development phase) 
are therefore not presented here although the Importance of such data Is fully 
acknowledged. 


5. UTILIZATION OF FINDINGS 

One of the problems of utilizing fault density as an Index of software 
reliability Is that It measures past events. One could even argue that a high 
number of faults detected during tests Indicates that few remain as the program 
enters the operational phase. However, most practitioners know that there Is a 
continuity to the fault detection process, and that a high rate of problems In 
one phase Is usually associated with an above average rate In succeeding phases. 
An Important Improvement In the Interpretation of the fault density data Is to 
segregate and emphasize faults that occur In the final test phase because these 
constitute a better Index of what might be expected In operation. Even with all 
these deficiencies, the data reviewed here make It unlikely that an average 
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flight control program Is free of faults when It enters the operational phase. 
The data also show a wide variation In fault densities, with flight control 
programs being frequently In the upper part of that range (cf. Table 4). This 
Indicates that much Improvement Is possible by Intensive use of existing 
methodology. 

The use of high order language seems to be an easily attainable means of 
Improving the reliability of flight control programs or others that are Involved 
In critical applications. Beyond this, the available Information does not 
Identify specific processes but It does Indicate In a general way that the 
adoption of disciplined development methodology that took place In the 1975 - 
1977 time frame redoes the fault content of programs (see Tables 3 and 5). 
That further Improvements through Intensive use of this methodology are possible 
seems to be borne out by the significantly lower fault density of the 
Goddard-SEL programs (see Table 1). That environment has encouraged full use of 
advances In software engineering, and several programs from this group were at 
the low limit of fault density encountered here (0.01$). Further studies to 
Isolate the benefits of Individual tools or methodologies are obviously 
Indicated. 

That software tools can play an Important part In the reduction of program 
faults before they reach test Is strongly Indicated by the high percentage of 
logic, data base, and data I/O faults (Table 7). In the Ames (Set 1) sample, 
these three categories accounted for over 75$ of all faults. Many of these can 
be avoided by the use of static analyzers, set/use listings, data dictionaries, 
and similar readily available tools. The high percentage of data 1/0 faults In 
flight control programs suggests emphasis on tools for mechanizing the review of 
data Interchange with a program. 


6. RECOMMENDATIONS 

Collection of failure rate data, particularly tied to execution time, Is 
essential In order to obtain a software reliability measure that can be directly 
compared with the FAA certification requirements. Many examples for the 
collection of such data are available CHECH77, MUSA79D. 


The data should be segregated by test phases or time In operational status so 
that Improvements In reliability with time can be determined. To permit the 
Identification of beneficial methodologies and tools, these data should be 
supplemented by a full description of the development, test, and operational 
environments. 

The classification of faults and errors needs to be considerably modified In 
order to address flight control concerns. It must provide a basis for 
associating error types (by type of manifestation and severity) with fault types 
(logic, etc. as used above) so that emphasis can be placed on eliminating those 
faults that have the greatest potential for causing catastrophic failures. The 
recording of the function (e.g., computational routine, I/O routine, scheduler) 
that Is affected by each failure Is also of Importance. The full benefits of 
failure reports are seldom achieved If a thorough analysis Is not performed at 
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the time the failure occurs and the correction Is made. At that time, the 
Information Is fresh and Individuals Involved In the detection and correction 
process are available. Much more effort Is needed In both developing and 
Implementing Improved methods of failure reporting. A more meaningful 
Identification of the development phase during which the fault arose may also be 
achieved by those efforts. 

Attention has recently been called to the effect of workload (the fractton of 
computer processing capability utilized) on failure frequency of computers 
CCAST32, t YER82U • This suggests that (a) workload be considered as a failure 
Inducing element In any reliability analysis to demonstrate compliance with the 
FAA requirements, and (b) that the prevailing workload be recorded In failure 
reports to permit further study of this phenomenon. 

It Is recognized that most flight control software failures will not have 
catastrophic effects on the safety of the aircraft. Yet, the relationship 
between frequency and severity of failures Is largely unexplored. An 
investigation of failures In the Bell No. 4 Electronic Switching System (ESS) 
suggests that there Is an Inverse relation between frequency and severity (1. 
e., failures that caused the longest service Interruption occurred least 
frequently) CDAVI813. Whether this holds irue for flight control software 
failures needs to be established. If such a relation exists In a general sense, 
then certification could be based on the frequency of occurrence of mild 
failures, a much more readily observable quantity. 

To meet the needs of reliable software for critical applications In the 
Immediate future, the application of fault tolerance techniques appears 
essential, not as an alternative to efforts at Improving the quality of 
Individual programs but rather as a supplement to such efforts. Several methods 
for Implementing software fault tolerance have been described In the literature 
[RAND75, CHEN78, HECH793. All require multiple, Independent versions of a 
program but there are differences In the manner In which the Independence Is 
achieved and In the error detection mechanism. Even with allowance for less 
than perfect operation of the fault tolerance provisions, these approaches can 
reduce the software failure rate by several orders of magnitude. Thus, It may 
be possible to demonstrate compliance with the FAA requirements by showing that 
an Individual program has a catastrophic failure rate of less than IE-6, and 
that fault tolerance provisions reduce the system catastrophic failure rate to 
less than IE-9 per flight hour. 
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APPENDIX — DATA SUMMARY 


A summary of the data utilized In the analysis of fault densities (Section 3) Is 
shown In Table A. The 28 projects listed In the table were selected from about 
50 projects which were originally Included In the study. The deleted projects 
had Incomplete data In one or more of the following categories which were 
essential for the principal analyses carried out In this report: 

Year of start 

Number of executable statements 

Number of faults 

Primary programming language 

As Is apparent In the table, the selected programs were frequently deficient In 
other data, and this caused some secondary analyses In the report to be based on 
a smaller population. 

The following explanations apply to Table A: 

PROJ NAME - Project designation, not necessarily descriptive. Intended only to 
show the general origin of the software. The numbers appended to the NASA 
Goddard data correspond to the Project Number In the SEL data base. 

ORG - Origin of the data: 1 = NASA Ames dataset 1, 2 = NASA Ames dataset 2, 

N = NASA Goddard Software Engineering Laboratory 

START YR - Year of start of the prgram development 

END YR - Year program development was completed 

PGMR HRS - Programmer hours devoted to development, an entry of 0 Indicates 
missing data 

MGMT, OTH HRS - Management and other classification hours devoted to 
development, an entry of 0 Indicates missing data 

EXEC STMTS - Equivalent executable assembly statements as explained on p. 3 
FAULTS - Number of faults reported for this project 

PRIM LANG - Primary programming language. The following abbreviations are used: 
ASM - assembly language, COB - COBOL, FORT IV - FORTRAN IV, SFOR - S-FORTRAN 

% - Percent of executable statements In primary language, an env. y of 0.00 
Indicates missing data, an entry of 99.99 Indicates that all statements w ere In 
the primary language 

SEC LANG - Secondary Language. Abbreviations as under PRIM LANG; UNK Indicates 
that It Is not known whether a secondary language was used 
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% - As under PRIM LANG 

FAULT G£NS - Fault density as explained on p. 3 


TABLE A - BATA SUMMARY 


PROJ NAME 

ORB START 
YR 

END 

YR 

P6MR HRS 

HGMT. 
OTH HRS 

EXEC 

STMTS 

FAULTS 

PRIM LANG 

X SEC LANG 

l 

FAULT 

DENS, 

PROJ X ADCU 

1 

1977 

1981 

57800 

0 

19482 

309 

ASM 

99.99 NONE 

0.00 

0.01586 

PROOl X - CLOBBER ANAL MODULE 

1 

1979 

1981 

0 

0 

66866 

141 

COB.SFOR 

0.00 ASM 

0.00 

0.00211 

PROJ X-HBPSiMJSSIDN DATA BRER) 

I 

1979 

1981 

21250 

0 

157535 

252 

FORT VJ 

99.99 NONE 

0,00 

0.00160 

PROJ Y - COMMAND F DISPLAY 

i 

1972 

1975 

0 

0 

6640 

42 

ASM 

69.30 JOVIAL 

30.70 

0.00630 

PROJ Y - NAVIGATION 

i 

1972 

1975 

0 

0 

32980 

316 

JOVIAL 

99.99 NONE 

0.00 

0.00960 

PROJ V - EXECUTIVE 

i 

1972 

1975 

0 

0 

6466 

151 

ASM 

99.99 NONE 

0,00 

0.02335 

PROJ Z - CAP PHASE I 

l 

1974 

1974 

0 

0 

3400 

165 

ASM 

79.35 FORTRAN 

20.64 

0.04850 

PROJ Z - MSAP PHASE I 

I 

1974 

1974 

0 

0 

600 

70 

ASM 

99.99 NONE 

0,00 

0,11666 

PROJ Z - SAP PHASE I 

j 

1974 

1974 

0 

0 

4450 

162 

ASM 

0.00 UNX 

0.00 

0.03640 

PROJ Z - EXEC PHASE I 

i 

1974 

1974 

0 

0 

12572 

163 

ASM 

99.99 NONE 

0.00 

0,01296 

PROJ Z - CAP PHASE II 

l 

1978 

1978 

0 

0 

16048 

258 

ASH 

85.89 FORTRAN 

14.41 

0.01610 

PROJ Z - SAP PHASE 11 

i 

1978 

1978 

0 

0 

4450 

143 

ASM 

99,99 NONE 

0,00 

0.03213 

PROJ Z - EXEC PHASE II 

i 

1978 

1979 

0 

0 

1228 

64 

ASM 

99.99 NONE 

0.00 

0.05211 

PROJ Z - SIMULATOR PHASE II 

l 

1978 

1979 

0 

0 

20618 

192 

ASM 

92.09 FORTRAN 

7.91 

0.00930 

PROJ Z - SUPPORT SN PHASE II 

l 

1978 

1979 

0 

0 

38218 

72 

ASM 

94.96 FORTRAN 

5.03 

0,00190 

PriV E (FLT. CONTR. ) 

n 

4 

1978 

1978 

0 

0 

44400 

381 

AED 

99.99 NONE 

0.00 

0.00860 

PROJ D (FLT. CONTR.) 

2 

1978 

1978 

0 

0 

43500 

78 

AED 

99.99 NONE 

0.00 

0.00180 

AEM - MANPOWER ALLOC #2 

N 

1977 

1978 

89115 

47855 

26488 

112 

FORTRAN 

84.00 ASM 

16,00 

0.00423 

ISEEB-INTNATL SUN EARTH XPR 15 

N 

1977 

1977 

129299 

37096 

122718 

88 

FORTRAN 

87.00 ASM 

13.00 

0.00070 

PAS-PANORAM ATTITUDE SCAN *6 

N 

1977 

1977 

128522 

72188 

198965 

21 

FORTRAN 

86.00 ASM 

14.00 

0.00010 

SEASAT #10 

N 

1977 

1978 

109565 

47820 

109147 

59 

FORTRAN 

76.00 ASM 

24.00 

0.00050 

FOXPRP-SHM FOCUS PROC 135 

N 

1978 

1979 

19205 

11278 

16997 

17 

FORTRAN 

72.00 ASH 

28.00 

0.00100 

DEA-DYNAMICS XPLR A *36 

N 

1980 

1980 

149476 

73735 

140812 

224 

FORTRAN 

87.00 ASM 

13.00 

0.00159 

DEB-DYNAMICS XPLR B *37 

N 

1980 

1980 

134639 

77997 

128444 

177 

FORTRAN 

85.00 ASM 

15.00 

0.00138 

DESIM-DYNAMICS XPLR SIM *38 

N 

1980 

1980 

31638 

24964 

22410 

17 

FORTRAN 

99.99 NONE 

0.00 

0.00080 

DEBET-BEB DETERMINISTIC- *40 

N 

1980 

1980 

34532 

18750 

13829 

49 

FORTRAN 

79.00 ASM 

21.00 

0.00290 

MASNRT-MASSAT NR REAL TIME *4? 

N 

1980 

19B0 

14023 

3885 

28900 

2 

FORTRAN 

85.00 ASM 

15.00 

0.00010 

HASIRC-HASSAT IR CAL I BRA *54 

N 

1980 

1980 

5182 

3408 

2920 

3 

FORTRAN 

99.99 NONE 

0.00 

0.00100 


